Jupyter Notebook In Depth

Today we are going to dive into some of the interesting features of IPython and the Jupyter notebook, which are useful for a number of daily tasks in data-intensive science.

We will work through these features "live"; feel free to type along with me as we go!

Outline

  • IPython command-line vs. Jupyter notebook
  • Input/Output History
  • Tab Completion
  • Getting help and accessing documentation
  • Useful Keyboard Shortcuts
  • Magic Commands
  • Shell commands
  • Interactivity with ipywidgets

IPython Command Line and Jupyter Notebook

Launching the IPython Shell

If you have installed IPython correctly, you should be able to type ipython in your command prompt and see something like this:

IPython 4.0.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
In [1]:

With that, you're ready to follow along.

Launching the Jupyter Notebook

The IPython notebook is a browser-based graphical interface to the IPython shell, and builds on it a rich set of dynamic display capabilities. As well as executing Python/IPython statements, the notebook allows the user to include formatted text, static and dynamic visualizations, mathematical equations, javascript widgets, and much more. Furthermore, these documents can be saved in a way that lets other people open them and execute the code on their own systems.

Though the Jupyter notebook is viewed and edited through your web browser window, it must connect to a running Python process in order to execute code. This process (known as a "kernel") can be started by running the following command in your system shell:

$ jupyter notebook

This command will launch a local web server which will be visible to your browser. It immediately spits out a log showing what it is doing; that log will look something like this:

jakevdp$ jupyter notebook
[I 11:02:44.237 NotebookApp] The port 8888 is already in use, trying another port.
[I 11:02:44.251 NotebookApp] Serving notebooks from local directory: /Users/jakevdp/github/uwseds/LectureNotes-Autumn2017/06-Jupyter-Notebook-In-Depth
[I 11:02:44.251 NotebookApp] 0 active kernels 
[I 11:02:44.251 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/?token=cf6e2acfe3bcf9f2f1529dedd59515560ae004ddeba6b882
[I 11:02:44.251 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation)

At the command, your default browser should automatically open and navigate to the listed local URL; the exact address will depend on your system. If the browser does not open automatically, you can open a window and copy this address (here http://localhost:8888/ with an authentication token) manually.

Input/Output History

A useful feature of IPython/Jupyter is the storage of input and output history.

Terminal Only

There are a few useful shortcuts that can be used only in the IPython terminal. We will demonstrate the following in the terminal:

  • up arrow for history
  • partial completion with up arrow
  • reverse search with ctrl-r

Terminal and Notebook

  • Previous results can be obtained using underscores: number of underscores is the number of the previous command:

In [1]:
1 + 1


Out[1]:
2

In [2]:
_ * 3


Out[2]:
6

In [3]:
_ + __


Out[3]:
8

In [4]:
_ + __ + ___


Out[4]:
16

Beyond three underscores, you can use underscore followed by a number to reference cell number:


In [5]:
_2


Out[5]:
6

More useful often is the Out array, which stores all previous results:


In [6]:
Out


Out[6]:
{1: 2, 2: 6, 3: 8, 4: 16, 5: 6}

In [7]:
Out[3]


Out[7]:
8

Similarly, you can access the In array to see the code history:


In [8]:
In


Out[8]:
['', '1 + 1', '_ * 3', '_ + __', '_ + __ + ___', '_2', 'Out', 'Out[3]', 'In']

To see all history at once, use the %history magic command (more on magic commands below):


In [9]:
%history


1 + 1
_ * 3
_ + __
_ + __ + ___
_2
Out
Out[3]
In
%history

Tab Completion

The feature of Jupyter that I use the most frequently is perhaps the tab completion functionality.

Tab completion works for finishing built-in commands:


In [10]:
ra


Out[10]:
range(0, 4)

It works for variables that you have defined:


In [13]:
ThisIsALongVariable = 3.14

In [14]:
ThisIsALongVariable


Out[14]:
3.14

It works for importing packages:


In [15]:
import pa

In [ ]:

It works for finding attributes of packages and other objects:


In [ ]:
numpy.ra

In [ ]:

Accessing Help and Documentation

After tab completion, I think the next most useful feature of the notebook is the help functionality.

One question mark after any valid object gives you access to its documentation string:


In [16]:
numpy.arange?

In [ ]:

Two question marks gives you access to its source code (if the object is implemented in Python):


In [17]:
import pandas
pandas.DataFrame??

In [ ]:

This even works with user-defined functions:


In [18]:
def add_two_numbers(a, b):
    """Add two numbers and return the result"""
    return a + b

In [19]:
add_two_numbers??

In addition, you can use a single question mark with asterisks to do a wildcard match:


In [20]:
numpy.*cos*?

In [21]:
numpy.arccosh?

If you are curious about the call signature for a funciton, you can type shift tab within the open-closed parentheses to see its argument list:


In [ ]:
numpy.arccosh()

Hitting shift tab multiple times will give you progressively more information about the function:


In [ ]:
numpy.arccosh()

Using a combination of these, you can quickly remind yourself of how to use various funcitons without ever leaving the terminal/notebook.

Useful Keyboard Shortcuts

One of the keys to working effectively with Jupyter is learning your way around the keyboard. Note: some of the shortcuts below will only work on Linux and Mac; many will work on windows as well

Terminal shortcuts

If you are familiar with emacs, vim, and similar tools, many of the terminal-based keyboard shortcuts will feel very familiar to you:

Keystroke Action
Ctrl-a Move cursor to the beginning of the line
Ctrl-e Move cursor to the end of the line
Ctrl-b or left-arrow Move cursor back one character
Ctrl-f or right-arrow Move cursor forward one character

Text Entry

Keystroke Action
backspace Delete previous character in line
Ctrl-d Delete next character in line
Ctrl-k Cut text from cursor to end of line
Ctrl-u Cut all text in line
Ctrl-y Yank (i.e. Paste) text which was previously cut
Ctrl-t Transpose (i.e. switch) previous two characters

Command History

Keystroke Action
Ctrl-p or up-arrow Access previous command in history
Ctrl-n or down-arrow Access next command in history
Ctrl-r Reverse-search through command history

Miscellaneous

Keystroke Action
Ctrl-l Clear terminal screen
Ctrl-c Interrupt current Python command
Ctrl-d Exit Jupyter session

Notebook Shortcuts

Depending on your operating system and browser, many of the navigation and text-entry shortcuts will work in the notebook as well. In addition, the notebook has many of its own shortcuts.

First, though, we must mention that the notebook has two "modes" of operation: command mode and edit mode.

  • In command mode, you are doing operations that affect entire cells. You can enable command mode by pressing the escape key (or pressing ctrl m). For example, in command mode, the up and down arrows will navigate from cell to cell.
  • In edit mode, you can do operations that affect the contents of a single cell. You can enable edit mode by pressing enter from the command mode. For example, in edit mode, the up and down arrows will navigate lines within the cell

To get a listing of all available shortcuts, enter command mode and press "h"

Magic Commands

IPython & Jupyter extend the functionality of Python with so-called "magic" commands: these are marked with a % sign. We saw one of these above; the %history command.

Magic commands come in two flavors: line magics start with one percent sign, and cell magics start with two percent signs.

We'll go through a few examples of magic commands here, but first, using what you've seen above, how do you think you might get a list of all available magic commands? How do you think you might get help on any particular command?


In [24]:
%history


1 + 1
_ * 3
_ + __
_ + __ + ___
_2
Out
Out[3]
In
%history
range(4)
import numpy
numpy.ra
ThisIsALongVariable = 3.14
ThisIsALongVariable
import pandas
numpy.arange?
import pandas
pandas.DataFrame??
def add_two_numbers(a, b):
    """Add two numbers and return the result"""
    return a + b
add_two_numbers??
numpy.*cos*?
numpy.arccosh?
numpy.arccosh()
numpy.arccosh()
%history

Profiling with timeit

For example, here's the %timeit/%%timeit magic, which can be very useful for quick profiling of your code:


In [25]:
%%timeit
val = 0
for i in range(2000000):
    val = val + i


142 ms ± 3.73 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [26]:
%timeit numpy.arange(2000000).sum()


6.67 ms ± 691 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Creating a file with %%file

Sometimes it's useful to create a file programatically from within the notebook


In [27]:
%%file test.py
x = 4
print("the value of x is", x)


Overwriting test.py

Running a script with %run


In [28]:
%run test.py


the value of x is 4

Controlling figures: %matplotlib

You can use the %matplotlib function to specify the matplotlib backend you would like to use. For example:

  • %matplotlib by itself uses the default system backend
  • %matplotlib inline creates inline, static figures (great for publication and/or sharing)
  • %matplotlib notebook creates inline, interactive figures (though in my experience it can be a bit unstable)

In [29]:
%matplotlib inline

In [30]:
import matplotlib.pyplot as plt
plt.plot([1, 2, 3])


Out[30]:
[<matplotlib.lines.Line2D at 0x11a554550>]

matplotlib notebook will give you an interactive figure:


In [31]:
%matplotlib notebook
plt.plot([1, 2, 3])


Out[31]:
[<matplotlib.lines.Line2D at 0x11a65af28>]

Help functions and more info

  • Tab completion after % works!
  • The %magic function will tell you all about magic commands
  • The %lsmagic function will list all available magic commands
  • Remember that the ? can be used to get documentation!
  • Though we won't cover it here, it is possible to create and activate your own magic commands

In [32]:
%lsmagic


Out[32]:
Available line magics:
%alias  %alias_magic  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %colors  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %popd  %pprint  %precision  %profile  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

Available cell magics:
%%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

Automagic is ON, % prefix IS NOT needed for line magics.

Shell Commands

Jupyter is meant to be an all-purpose scientific computing environment, and access to the shell is critical. Any command that starts with an exclamation point will be passed to the shell.

Note that because windows has a different kind of shell than Linux/OSX, shell commands will be different from operating system to operating system.

All the commands you have learned previously will work here:


In [33]:
!ls


Jupyter Notebook In Depth (Completed).ipynb
Jupyter Notebook In Depth.ipynb
LorenzSystem.ipynb
WidgetsDemo.ipynb
test.py

In [34]:
!pwd


/Users/karla/CSE583/LectureNotes-Autumn2017/06-Jupyter-Notebook-In-Depth

In [35]:
!cat test.py


x = 4
print("the value of x is", x)

You can even seamlessly pass values to and from the Python interpreter. For example, we can store the result of a directory listing:


In [36]:
x = !cat test.py

In [37]:
x


Out[37]:
['x = 4', 'print("the value of x is", x)']

We can inject Python variables into a shell command with {}:


In [38]:
filename = 'test.py'

In [39]:
x = !cat {filename}

In [40]:
for i in range(10):
    !echo {str(i)}


0
1
2
3
4
5
6
7
8
9

With these tools, you should never have to switch from IPython to a terminal to run a command.

Rich Display

Many Python libraries are designed with Jupyter in mind, and use rich display hooks to improve the coding experience. One example we've seen is Pandas:


In [41]:
import pandas as pd

In [42]:
data = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

In [43]:
data


Out[43]:
A B
0 1 4
1 2 5
2 3 6

In [44]:
print(data)


   A  B
0  1  4
1  2  5
2  3  6

In [45]:
data._repr_html_??

If you want to do this sort of thing for your own object, you can create a class which defines the _repr_html_ method, and returns a string of HTML:


In [46]:
class RedText(object):
    def __init__(self, text):
        self.text = text
        
    def _repr_html_(self):
        return "<font color='red' size=24>" + str(self.text) + "</font>"

In [47]:
RedText('hello there')


Out[47]:
hello there

Exercise:

Try writing a class that, given a Python list, will display the contents of the list using formatted HTML (see example HTML lists at https://www.w3schools.com/html/html_lists.asp)

Recall that a Python list looks like x = ["a", "b", "c"]


In [48]:
class ListDisplay(object):
    def __init__(self, L):
        self.L = L
        
    def _repr_html_(self):
        output = '<ul>'
        for value in self.L:
            output += "<li>" + str(value) + "</li>"
        output += "</ul>"
        return output
    
my_list = [1, 2, 3]
ListDisplay(my_list)


Out[48]:
  • 1
  • 2
  • 3

IPython Widgets

One incredibly useful feature of the notebook is the interactivity provided by the ipywidgets package. You'll have to install this using, e.g.

$ conda install ipywidgets

You can find a full set of documentation notebooks here. We're going to walk through a quick demonstration of the functionality in WidgetsDemo.ipynb